Papers with long context processing
LUMINA: Long-horizon Understanding for Multi-turn Interactive Agents (2026.findings-acl)
Copied to clipboard
Amin Rakhsha, Thomas Hehn, Pietro Mazzaglia, Fabio Valerio Massoli, Arash Behboodi, Tribhuvanesh Orekondy
| Challenge: | Large language models struggle on multi-turn, long-horizon agentic problems that require skills such as planning, state tracking, and long context processing. |
| Approach: | They propose an oracle counterfactual framework for multi-turn problems that asks: how would an agent perform if it could leverage an or acle to execute a specific skill? |
| Outcome: | The proposed framework allows for precise oracle interventions without confounding effects present in real-world benchmarks. |
Compressing Context to Enhance Inference Efficiency of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across various tasks. |
| Approach: | They propose a method that prunes redundancies in the input context to make the input more compact. |
| Outcome: | The proposed method reduces memory and inference time while maintaining comparable performance compared to full context. |
LLoCO: Learning Long Contexts Offline (2024.emnlp-main)
Copied to clipboard
Sijun Tan, Xiuyu Li, Shishir G Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph Gonzalez, Raluca Popa
| Challenge: | Large language models are still unable to handle long contexts due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. |
| Approach: | They propose a method to learn contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA. |
| Outcome: | The proposed model outperforms in-context learning while using 30 fewer tokens during inference and significantly reduces the cost of long document question answering. |